Human Document Classification Using Bags of Words

Humans are remarkably adept at classifying text documents into categories. For instance, while reading a news story, we are rapidly able to assess whether it belongs to the domain of finance, politics or sports. Automating this task would have applications for content-based search or filtering of digital documents. To this end, it is interesting to investigate the nature of information humans use to classify documents. Here we report experimental results suggesting that this information might, in fact, be quite simple. Using a paradigm of progressive revealing, we determined classification performance as a function of number of words. We found that subjects are able to achieve similar classification accuracy with or without syntactic information across a range of passage sizes. These results have implications for models of human text-understanding and also allow us to estimate what level of performance we can expect, in principle, from a system without requiring a prior step of complex natural language processing. Introspection suggests that for text understanding, humans use a form of representation that takes into account structural and layout information in addition to word-level information. This is particularly likely in a typical document classification task scenario where only limited time is available to choose, from a set of documents, those that are relevant to a certain query or interest. In such a scenario, it seems plausible that syntax and layout information, such as headlines or paragraph boundaries, may be important for performing the classification task. However, there is little systematic experimental work that directly tests this expectation; how would classification performance suffer if syntax and layout information were removed from text passages? Following standard practice in the field (Mitchell 1997), we refer to a syntax and layout free representation as a 'bag of words' (BOW). A BOW is a feature vector where each element in the vector indicates the presence (or absence) of a word. Our goal was to test whether human classification performance is compromised with a BOW based representation relative to normally structured text and whether increased time pressure on the task increases the need for a structured representation. Furthermore, we wanted to assess how classification performance with the two representations changes as a function of the number of words included in the passages. In order to probe these questions, we experimentally compared human document classification performance on fully structured documents with performance on a BOW representation, with and without time constraints …